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Abstract. We investigate the two-dimensional Hubbard model on the triangular lattice with 
anisotropic hopping integrals at half filling. By means of a self-energy functional approach, we 
discuss how stable the non-magnetic state is against magnetically ordered states in the system. 
We present the zero-temperature phase diagram, where the normal metallic state competes with 
magnetically ordered states with (n, tt) and (2n /3,2n /3) structures. It is shown that a non- 
magnetic Mott insulating state is not realized as the ground state, in the present framework, 
but as a meta-stable state near the magnetically ordered phase with (27r/3, 2n/3) structure. 



Geometrical frustration in strongly correlated electron systems has attracted current interest. 
One of the typical examples is an organic material k-(BEDT-TTF)2Cu2(CN)3 with the triangular 
lattice structure, where a non-magnetic insulating state is realized down to low temperatures 
T ~32mK[l]. This suggests that a novel spin-liquid insulating state is stable against magnetically 
ordered states, which stimulates further theoretical investigations on frustrated electron systems 
[2-9]. Although a non-magnetic insulating state was shown to be a probable candidate for 
the ground state of the Hubbard model on the triangular lattice [2-4,6], it has not been 
discussed how this state competes with an antiferromagnetic insulating state with 120° spin 
structure (120°-AFI) expected naively. Recently, it was claimed that the 120°-AFI state is, 
instead of the nonmagnetic insulating state, stabilized in the strong coupling regime by means 
of variational Monte Carlo (VMC) simulations [7]. However, this method may not deal with 
non-local correlations properly, which should be important in the two-dimensional frustrated 
systems. Therefore, it is highly desirable to deal with intersite correlations as well as onsite 
correlations on an equal footing in order to clarify the ground-state properties of the frustrated 
systems. 

Motivated by this, we study the two-dimensional half-filled Hubbard model with geometrical 
frustration. The model Hamiltonian is given by TC = Ho(t) + 7i'(U) with 

^o(t) = ^^trr'C^CrV, (1) 
rr' a 

W(U) =^f/n rT n ri , (2) 



and t (U) is the parameter-matrix of the one-particle (two-particles) term, where cj<j(c rcr ) creates 
(annihilates) an electron with spin <r(=f, J.) at site r, and n ra is the number operator. Here, 
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Figure 1. (a) [(b)] Sketch of the 
structure of the model Hamiltonian (a 
topologically equivalent structure of the 
model). 



t Yr i = t(t') is the nearest-neighbor (second nearest-neighbor) hopping, which is schematically- 
shown in Fig. 1 (a), and U the on-site Coulomb interaction. We also give a sketch of the 
square lattice with specific diagonal hopping in Fig. 1 (b), which is topologically equivalent to 
the triangular lattice with anisotropic hopping. When t = t', the system is equivalent to the 
Hubbard model on the triangular lattice. In the following, we set t = 1 as the energy unit. 

To study the ground-state properties of the system, we make use of a self-energy functional 
approach (SFA), which is based on the Luttinger-Ward variational principle [10]. It provides 
us with a non-perturbative treatment of strong correlations, and an efficient and tractable way 
to investigate the metal-insulator transition (MIT) with and without symmetry breaking [3, 11- 
13]. An advantage in this method is to take into account intersite correlations as well as local 
electron correlations on an equal footing, which enables us to discuss the competition between 
the nonmagnetic insulating state and the magnetically ordered states. 

In this method, the ground potential Q is given as a function of a reference self-energy S(t'), 

fi[£(t')] = fl(t') +Trln{- [u + n - t - E(t')] - Tr In j- [u + \i - t' - E(t')] _1 } (3) 

where Jl(t') is the ground potential for a reference system with the Hamiltonian H re f = 
7io(t') + 7i'(U). We note that the Hamiltonian of the reference system is defined by the 
substitution of the parameter- matrix t' for t in the original Hubbard Hamiltonian, where we 
should keep U unchanged. The variational condition dQ[S(t')]/dt' = provides us with an 
approximate self-energy which properly describes physical properties of the original model. 
When one applies the SFA to the present system, a cluster Anderson impurity model is one 
of the most appropriate reference systems. It is described by the following Hamiltonian, 

( Nb 

Wo(t') = \ EE *rr'4+R«x C r'+R,- + £ "r«L°£ra 

R ^ rr' cr £=i r cr 

+ E E E V M+n,a a er'a + H.c.) + £ exp(ir • Q)H Q • S r+R L , 

£=1 rr' cr r ' 

where ^ r (X^r) sum s up sites (clusters) in the cluster (the whole lattice) and {t£ r /, e" r , Vg, Hq} 
are variational parameters. a l r cr( a ercr) creates (annihilates) an electron with spin a at the 
£(= l,-- - , JVj,)th site in the effective bath, which is connected to the original lattice at site 
r. In order to deal with magnetically ordered states, we introduce the effective magnetic field 
Hq in the reference system. Here, S r = ^cJ 7 a 77 'C r7 ', where a is the Pauli matrix. In this study, 
we mainly discuss the triangular lattice, t = t' . In the following, to investigate the MIT of the 
half-filled Hubbard model, we examine a free energy F = Q — fiN with an additional variational 
condition dF/d/j, = 0, where fi is the chemical potential and iV is the total number of electrons. 

Before addressing a magnetic instability, we first consider the zero-temperature properties in 
the paramagnetic state. This might be important at very low temperatures since magnetically 
ordered states become unstable in two dimensions. Here, we use a two-site cluster with Hq = 




Figure 2. (a) [(c)] Lattice structure for two-(three-)site clusters, which are represented by 
rectangles (triangles), (b) [(d)] Cluster impurity model to describe a magnetically ordered state 
in the original model, where arrows indicate the direction of the effective magnetic field. Dark 
(white) circles correspond to the impurity-(bath-)site. 




Figure 3. (a) Free energy F as a function of the variational parameter V for t' = 1. Triangles 
(crosses) correspond to the stationary points for the paramagnetic insulating (paramagnetic 
metallic) states, (b) Free energies F for several states as a function of U when t' = 1. The inset 
shows an enlarged picture around U ~ 9.5. 



as a reference system, which is illustrated in Fig. 2 (b). The obtained free energy is shown 
in Fig. 3 (a). In the small U region (U < 7.8), we find one stationary point with a large 
V, which should represent the paramagnetic metallic (PM) state. The increase in U gradually 
changes the stationary point and decreases its variational parameter V, which implies that the 
heavy quasi-particle state is realized in the case U < 7.8. On the other hand, the paramagnetic 
insulating (PI) state appears in the strong coupling region (U > 10.3), which is shown as another 
stationary point in the figure. Thus, these two phases compete with each other and a first-order 
MIT between the PI and the PM states occurs around U ~ 9.6 as far as the paramagnetic states 
are concerned. 

However, this result does not necessarily imply that the PI state is stable against magnetically 
ordered states at zero temperature. To make this clear, we discuss the magnetic instability in the 
system. Potential candidates for the ordered states are the (ir, 7r)-antiferromagnetic insulating 
(AFI) state and the 120°-AFI state. By using two kinds of the reference systems illustrated 
in Figs. 2 (b) and (d), we obtain the free energies for paramagnetic and magnetically ordered 
states, as shown in Fig. 3 (b). We can see that the PM state is stable in the small U region. On 
the other hand, it is found that in the strong coupling limit, the 120°-AFI state is stabilized, 
which is consistent with the results for the Heisenberg model. At U c ~ 8.4, the free energies 
for these states cross each other, suggesting the first-order MIT accompanied by symmetry 
breaking. We also find that the non-magnetic insulating state is no longer realized as the 
ground state within the present approach. Nevertheless in the PM region (7.7 < U < 8.4), the 
non-magnetic insulating state exists as a meta-stable state. Therefore, if the interacting system 



Figure 4. Phase diagram of the half-filled 
Hubbard model. White region represents the PM 
phase, and light (dark) gray region the (tt,tt)- 
AFI [120°-AFI] phase. The thick dashed line 
indicates the boundary where the meta-stable PI 
state disappears with decreasing U. 

is adiabatically cooled down, the meta-stable non-magnetic insulating state could emerge down 
to zero temperature. 

By performing SFA calculations with several values of t', we obtain the ground-state phase 
diagram of the half-filled Hubbard model, which is shown in Fig. 4. The PM state is stable in 
the weakly correlated region with t' ^ 0. In the strongly correlated region, the (it, 7r)-AFI state 
competes with the 120°-AFI state and the phase transition occurs at t' c ~ 0.8. These results 
are consistent with those obtained by the VMC calculations [7]. In Fig. 4, we show the region 
where the non-magnetic insulating state is not realized as the ground state, but appears as a 
meta-stable state. 

In summary, we have investigated the half-filled Hubbard model on the triangular lattice with 
anisotropic hopping integrals by means of a self-energy functional approach. We have clarified 
how the normal metallic state competes with the magnetically ordered states with (it, it) and 
(27r/3, 27r/3) structures. When t ^ t', another type of magnetically ordered state with an 
incommensurate wave vector Q might be also possible [14], which has not been addressed in 
this paper. Therefore, it remains an important problem to explore the stability of such ordered 
states, which is now under consideration. Furthermore, it has been suggested that in the square- 
lattice model with next-nearest hopping, which is another important frustrated system slightly 
different from the present one, the collinear ordered phase with (ir, 0) or (0, ir) is stabilized 
for large frustration [5, 9]. Comprehensive investigations including such frustrated systems arc 
important, which are left for the future study. 
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